1. Field of the Invention
This invention relates to the cataloguing multimedia data and including storage and retrieval mechanisms.
2. Background
Increasingly, computer systems are being used to present multimedia material. Such material is usually in the form of text, graphics, video, animation, and sound. Two or more of these data types are usually combined to form the multimedia data presented by the computer system. A computer system that is used to present multimedia material is called a multimedia system. A problem with prior art multimedia systems is an inability to search and retrieve multimedia data.
One prior art multimedia system uses a disk operating system that includes a file system for storing and retrieving files containing multimedia data. The file system catalogues the files based on the names given to the files. The file system can be used to retrieve a file that contains multimedia data based on the file""s name. Other than the extent to which the file name identifies content, the file system does not provide the ability to retrieve multimedia information based on the content of the data. The search system provided by a file system is therefore inadequate to search with greater detail than that provided in a file name. A number of prior art multimedia systems are described below.
A search system is described in U.S. Pat. No. 5,241,671, Reed et al., issued on Aug. 31, 1993 relates to a multimedia system that includes a database that is comprised of words, phrases, numbers, letters, maps, charts, pictures, moving images, animations, and audio information. A search capability is provided that provides a series of entry paths for locating information in the database. An entry path allows a user to enter a search request that consists of a set of valid terms or stop terms. A stop term is a term that exists on a stop term list and may be the words xe2x80x9cthexe2x80x9d or xe2x80x9caxe2x80x9d, for example. Valid terms are linked to related terms using a stem index. A stem index contains a root term and a set of stems for each term that is related to the root word. For example, the word leaf is linked to terms xe2x80x9cleavesxe2x80x9d and xe2x80x9cleafingxe2x80x9d.
The creating and displaying of navigators for locating and accessing visual/image information is described in U.S. Pat. No. 5,123,088, Kasahara et al., issued on Jun. 16, 1992. Image information is categorized and linked in a circular list and ordered according to their attributes. When an image unit is displayed, the linked image units can be displayed as reduced images, or navigators. A user can navigate through a network of linked image units by selecting the desired navigator.
A system for database retrieval wherein entries in different databases are retrieved by a process of matching key words of the databases is described in U.S. Pat. No. 5,210,868, Shimada et al., issued on May 11, 1993. Examples of two such databases are a mapping database and a customer attribute database. A dictionary is used to separate a keyword from a first database into common and proper noun subparts. Common and proper noun synonyms are inferred according to a set of rules. The synonyms are combined using a combination rule and then compared with keywords in a second database to generate a final matching result.
A system for handling multimedia using entity and relation objects is described in U.S. Pat. No. 5,278,946, Shimada et al. issued on Jan. 11, 1994. An entity object defines methods and properties for entities such as a building, road, railroad, and boundary. A relation object defines method and properties for relationships between entities. A user model and system model can be coupled to generate a digest of multimedia data.
A system for storing and retrieving digital images is described in U.S. Pat. No. 5,493,677, Balogh et al., issued on Feb. 20, 1996. A caption or other metadata can be associated with a digital image. A natural language capability removes ambiguities from the metadata input by a user prior to its storage. The natural language capability determines matches between a user query and the stored metadata. The system allows a user to select an image, review licensing terms for the selected image, and order the image.
A repetitive analysis event system that accesses data using a time-based number is described in U.S. Pat. No. 5,414,644, Seaman et al., issued on May 9, 1995. The system uses an information library that consists of visual data storage and a textual database for storing written descriptions of the visual data and a glossary of keywords that identify repetitive events or behavior. A behavioral label is used to define a behavioral activity. A series of images or video clips are associated with the behavioral label. A user can retrieve images by identifying a subject, a behavioral activity, or other type of descriptive text. A chronological timeline is used to control the order in which the images are displayed. That is, the images are displayed in sequential order using the time-based number.
A knowledge based information retrieval system is described in U.S. Pat. No. 5,404,506, Fujisawa et al., issued on Apr. 4, 1995. The system provides a visual interface for local searching and a natural language interpreter for global search. The natural language interpreter is used to infer the meaning of a noun phrase or a nominal phrase. The inferred meaning is used to retrieve information.
The search capabilities in the patents identified above do not provide an ability to catalogue multimedia data such that it is available for use across systems or applications. There is no ability to create a general catalogue and index for searching a catalogue that can be used for the storage and retrieval of multimedia data by multiple applications. The storage approach used in the prior art is designed to accommodate a particular system""s needs. A number of other approaches are described, but in these too, the index capabilities are designed for use with a particular system.
An indexing capability designed for use with a hypertext nodal network is described in U.S. Pat. No. 5,408,655, Oren et al., issued on Apr. 18, 1995. The set of indexing terms generated using the hypertext nodal network are compared with each of the nodes in the database to identify a set of index terms for each node (i.e., document index terms). A set of index terms are associated with an option or criterion (i.e., option index terms) that can be user-selected from a menu. A hypertext nodal network is needed to use the indexing capability in this case.
An index is described in U.S. Pat. No. 5,121,470, Trautman, issued on Jun. 9, 1992 describes an interactive record system that automatically indexes data obtained from multiple sources. The invention has application in the medical care field. The data is indexed along one or more dimensions (e.g., time). Data events are identified for the indexed data by distinguishing sets of data into given intervals. Data objects are associated with the data events. The events and associated data objects are displayed. Actuators are associated with the data objects to allow the objects to be manipulated. Data events and dimensional criteria is needed to use this indexing scheme.
A system for identifying and displaying an image that is selected based on user input is described in U.S. Pat. No. 5,010,500, Makkuni et al., issued on Apr. 23, 1991. Gestures input using a mouse are used to identify an image having features that resemble the input. Multimedia data associated with a portion of the image can be activated by selecting the image portion. When a selection is made, a menu can be displayed for user selection. Data is indexed based on actual portions of images.
A system that creates an index for frame sequences in a motion image is described in U.S. Pat. No. 5,428,774, Takahashi et al., issued on Jun. 27, 1995. Each record in the index has an associated retrieval key. The initial and final positions of a frame sequence are designated in an index record. Records are retrieved from the index file based on the retrieval key. The retrieved records are arranged along a time axis based on the initial and final positions. Data (i.e., frame sequences of a motion picture) is indexed based on a time sequence of frames of the data.
A system that uses keywords to locate and retrieve higher level records is described in Kuga et al., U.S. Pat. No. 5,280,573, issued on Jan. 18, 1994. Each of a plurality of higher level records contain different types of information associated with a keyword. Such higher level records may contain usage, synonym, and meaning information associated with a keyword, for example.
A system for storing images and audio that can be used to create an audio-visual presentation is described in Beitel et al., U.S. Pat. Nos. 5,119,474 and 5,274,758, issued on Jun. 2, 1992 and Dec. 28, 1993, respectively. The system includes the following software components: library editor; image editor; digitize editor; convert editor; audio editor, and story editor. The image editor can edit an image (i.e., add text and graphics to an image). The digitize and audio editors convert analog data to digital data. The convert editor is used to convert images to a form that is usable by the system. Images and audio data are assembled into a presentation using the story editor. The library editor manages the storage, retrieval and processing of system objects (an object is an image, audio file or audio/visual presentation). The library editor maintains a library of files that contain an object.
The invention catalogues data such as multimedia data. A catalogue is a collection of one or more catalogue elements. An index is used to access a catalogue. An element of a catalogue has one or more attributes. An attribute provides information that can be used to search for, answer questions about, and navigate through a catalogue. An attribute of a catalogue element can be an element that has attributes. A catalogue element attribute that is an element is referred to as an attribute element. Attribute elements and attributes are used to build an index that can be used to facilitate catalogue access. Within a catalogue, smaller catalogues can be created by, for example, querying and user designation.
An analogy can be drawn between the multimedia catalogue of the invention and a card catalogue in a library. The card catalogue contains a single catalogue element (i.e., a card). The card catalogue is comprised of a plurality of catalogue element instances (i.e., a plurality of cards). Each card is associated with a book or some other material contained in the library. A card contains information or attributes (e.g., author, title, numbering system). The attributes are used to build an indexing system (e.g., numbering system) that can be used to facilitate a search through the catalogue. The card contains information or attributes that can be used to satisfy a query operation (e.g., select a card from the card catalogue) performed on the card catalogue.
The multimedia data catalogue used in the invention preferably consists of one catalogue element that is referred to as a phrase. A phrase is associated with a portion of multimedia data. A phrase has a plurality of attributes some of which are attribute elements. The attribute elements that are attributes of a phrase include keyword, person, image, video (e.g., documentary footage), proposed person, and proposed keyword. The keyword, person, image, proposed person and proposed keyword attribute elements can have attributes that are also attribute elements. For example, attribute elements that are attributes of the keyword attribute element include thesaural keyword, thesaural person, keyword, and type. An index is built on the attributes and attribute elements. The index can be used to navigate through the catalogue (e.g., search for phrases).
A set of catalogue elements can be identified by querying the attribute elements. A query operation can be performed on the attribute elements to examine other attribute elements associated with a catalogue element. A query operation identifies a set of cataloguing elements (e.g., phrases) that satisfy the criteria specified in the query. A set of cataloguing elements identified in a query are grouped as a segment. A segment is a container element that contains a collection of phrases. A user can also specify a collection of phrases that can be grouped as a segment. Segments can contain other groupings of phrases (e.g., phrases associated with a related input data portions). A segment can have their own attributes and attribute elements.
In addition to the catalogue and attribute elements, the invention associates a plurality of event elements with the input data. An event element is associated with an input data portion. An event element is a quality assurance event, for example, that marks an input data portion that has relevance to quality assurance data collection. Quality assurance events can be identified as a positive, negative, or informational event. A quality assurance event can also identify a fact or piece of information. An event element can also be associated with other elements such as a catalogue element (e.g., phrase) or a container element (e.g., segment).
Catalogue, attribute and event elements are instantiated and maintained using cataloguing and relationship management facilities. The cataloguing and relationship management facilities are provided to accept user input and generate elements and their attributes. The collection of element instances created using the cataloguing and relationship management facilities can be used as an interface by any system to access input data.
One attribute of a phrase element is referred to as a descriptive phrase. A user enters a descriptive phrase associated with an instance of the phrase element using the cataloguing facility interface. The cataloguing facility parses the descriptive phrase to identify existing attribute elements to associate with the phrase element. In addition, the descriptive phrase can be used to identify potential or proposed attribute elements such as proposed person and proposed keyword. Proposed attribute elements can also be entered by a user. A proposed attribute element is approved prior to its use as an actual attribute element.
The thesaural keyword and thesaural person attribute elements are examples of attribute elements that contain alternate information. Thesaural keyword and person attribute elements contain alternate information for the keyword and person attributes elements, respectively. For example, a keyword attribute element has a plurality of associated labels or values. A label associated with a keyword attribute element is supplied by a thesaural keyword attribute element. Thus, a thesaural keyword attribute element is an instantiation of a keyword attribute element.
Thesaural keyword attribute elements can further be used to supply a label in one or more languages for a keyword attribute element. That is, a keyword attribute element can have a plurality of labels in a plurality of languages. For example, a keyword attribute element can be associated with a plurality of thesaural keyword attribute elements each having a label, or value, represented in a language (e.g., English, French, German, and Russian). Each keyword attribute element having a plurality of labels in a language can have a preferred label in that language. Thus, a thesaural keyword attribute element can be an instantiation of a language thesaural keyword attribute element.
A type attribute element is used to classify keyword attribute elements. A type is associated with a keyword attribute element to classify the keyword attribute element. A hierarchy of type attribute elements can be used to classify instances of the type attribute element and their associated keyword attribute element instances. Similarly, a keyword attribute element is used to classify a thesaural keyword attribute element. A thesaural keyword attribute element is used to classify a language thesaural keyword attribute element.
Relationships can exist between two elements. A relationship can be xe2x80x9cis_axe2x80x9d, whole/part, or association. An xe2x80x9cis_axe2x80x9d relationship establishes an inheritance relationship between two or more elements. A whole/part relationship identifies the elements that are contained within another element. An association relationship is used to relate one element to another element.
The catalogue element(s) and associated attributes and attribute elements can be used to catalogue different sets of multimedia data. Further, within a catalogue, alternate expressions of content can be achieved using thesaural keywords and alternate thesauruses. Thus, the nature of the catalogue associated with multimedia data can be varied within a thesaurus (using thesaural keyword instances) or by using multiple thesauruses.
A phrase element has one or more associated keywords that identify content. Each keyword has one or more associated instances of thesaural keyword. Each thesaural keyword instance contains an alternate expression of content for its associated keyword. Thus, alternative content specifications can be created using alternative instances of thesaural keyword.
In addition, content associated with a catalogue can be varied using multiple thesauruses for a catalogue. A thesaurus can be defined using the keyword and type hierarchies formed from instances of keyword and type attribute elements. Each thesaurus can contain an alternative specification of content for a catalogue. That is, each thesaurus can contain instances of keyword and type attribute elements having an alternate content or concept. specification. Thus, content associated with a catalogue can vary with each instance of a thesaurus.